Analysis of Probabilistic Processes and Automata Theory

This chapter surveys some basic algorithms for analyzing Markov chains (MCs) and Markov decision processes (MDPs), and discusses their computational complexity. We focus on discrete-time processes, and we consider both finite-state models as well as countably infinite-state models that are finitely-presented. The analyses we will primarily focus on are hitting (reachability) probabilities and ω-regular model checking, but we will also discuss various reward-based analyses. Although it may not be evident at first, there are fruitful connections between automata theory and stochastic processes. Firstly, and not surprisingly, ω-automata play a naturally important role for specifying ω-regular properties of sample paths (trajectories) of stochastic processes. Computing the probability of the event that a random sample path satisfies a given ω-regular property constitutes the (linear-time) model checking problem for probabilistic systems. Secondly, it turns out that there are close relationships between classic infinite-state automatatheoretic models and classic denumerably infinite-state stochastic processes, even though these models were developed independently in separate mathematical communities. Roughly speaking, some classic stochastic processes share their underlying state transition systems with corresponding classic automata-theoretic models. Furthermore, exploiting these connections to automata theory is fruitful for the algorithmic analysis of such stochastic processes, and for their controlled MDP extensions. This holds even when the analyses are much simpler than model checking, such as computing (optimal) hitting probabilities. A number of important infinite-state stochastic models connected with automata theory can be captured as (restricted fragments of) recursive Markov chains and recursive Markov decision processes, which are obtained by adding a natural recursion feature to finite-state MCs and MDPs. Key computational problems for analyzing classes of recursive MCs and MDPs can be reduced to computing the least fixed point (LFP) solution of corresponding classes of monotone systems of nonlinear equations. The complexity of computing the LFP for such equations is a intriguing problem, with connections to several areas of research in theoretical computer science. D R A FT 2 K. Etessami